Second Order Derivatives for Network Pruning: Optimal Brain Surgeon

We investigate the use of information from all second order derivatives of the error function to perform network pruning (i.e., removing unimportant weights from a trained network) in order to improve generalization and increase the speed of further training. Our method, Optimal Brain Surgeon (OBS), is significantly better than magnitude-based methods, which can often remove the wrong weights. OBS also represents a major improvement over other methods, such as Optimal Brain Damage [Le Cun, Denker and Solla, 1990], because ours uses the full off-diagonal information of the Hessian matrix H. Crucial to OBS is a recursion relation for calculating H-1 from training data and structural information of the net. We illustrate OBS on standard benchmark problems — the MONK’s problems. The most successful method in a recent competition in machine learning [Thrun et al., 1991] was backpropagation using weight decay, which yielded a network with 58 weights for one MONK’s problem. OBS requires only 14 weights for the same performance accuracy. On two other MONK’s problems, our method required only 38% and 10% of the weights found by magnitude-based pruning. INTRODUCTION A central problem in machine learning is minimizing the system complexity (description length, VC-dimension, etc.) consistent with the training data. In neural networks this problem is usually cast as minimizing the number of connection weights. It is well-known that without such elimination of weights overfitting problems and thus poor performance on untrained